Skip to content

Drop the coreml_static_int8 operators job, which has never once finished - #21695

Merged
JakeStevens merged 1 commit into
mainfrom
shoumikhin/drop-coreml-static-int8-operators
Aug 11, 2026
Merged

Drop the coreml_static_int8 operators job, which has never once finished#21695
JakeStevens merged 1 commit into
mainfrom
shoumikhin/drop-coreml-static-int8-operators

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Addresses #21623.

What is broken

Every night, the CoreML backend workflow runs this job on a macOS runner:

test-backend-macos (coreml_static_int8, operators)

It has never finished. It runs for the full 180 minute limit, gets killed, and reports cancelled. That is about three macOS runner hours per night, every night, for no result at all, on a runner pool that is already the scarcest resource we have.

For comparison, in a recent nightly run:

job outcome
coreml / operators passed in 11m41s
coreml_static_int8 / models passed in 26m41s
coreml_static_int8 / operators killed at 3h00m50s

So it is specifically the combination of the heavier quantized flow with the much larger operators suite that does not fit.

Why we cannot just make it faster

The suite already runs with pytest -n auto, so it is using every core on the runner. There is no sharding support in _test_backend.yml to split it across several runners. Adding that is real work, and it is the right long term answer, but it is not something to land while the job is burning runner hours nightly.

The fix

Stop scheduling that one combination.

_test_backend.yml builds its matrix as every flow crossed with every suite, so today the only way to shape it is to change the flow list, which would also drop coreml_static_int8 / models, and that one passes and is worth keeping.

This adds an optional exclude input to _test_backend.yml that is wired straight into strategy.matrix.exclude. It defaults to an empty list, so nothing changes for any other backend. test-backend-coreml.yml then uses it to drop the single coreml_static_int8 plus operators pair.

After this change the nightly CoreML run is coreml / models, coreml / operators and coreml_static_int8 / models, all three of which actually complete.

When to revert this

When the operators suite can be sharded across runners, or when the quantized flow gets fast enough to fit in the limit. #21623 tracks that.

How this was verified

Two separate things needed checking: that the new input is inert for every other
caller, and that the exclude entry removes the job we mean and nothing else.

  • Inert by default. _test_backend.yml has 8 callers in total. With this
    branch's version of it, the 7 other callers still expand to every flow crossed
    with both suites. An exclude list that matches no combination is simply ignored by
    GitHub Actions, so the empty default is a no-op.
  • Correct entry. test-backend-coreml.yml only runs coreml_static_int8 on the
    nightly schedule, so a pull request run would not show the effect at all. It
    was therefore dispatched on a scratch branch with both flows forced on. The
    resulting job list was (coreml, models), (coreml, operators) and
    (coreml_static_int8, models). The (coreml_static_int8, operators) pair was
    gone and nothing else moved.

Overlap with other pull requests

.github/workflows/test-backend-coreml.yml is also edited by #21691, which adds a
schedule discriminator to the concurrency: block at the top of the file. This PR
adds a matrix exclude: entry further down. The hunks do not touch and the two
merge cleanly in either order.

Copilot AI lite review requested due to automatic review settings August 8, 2026 23:04
@pytorch-bot

pytorch-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21695

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit ad459d8 with merge base 9cd0c12 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@shoumikhin
shoumikhin force-pushed the shoumikhin/drop-coreml-static-int8-operators branch from f3e0024 to 3b0dd40 Compare August 10, 2026 05:51
Copilot AI review requested due to automatic review settings August 10, 2026 05:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 10, 2026 18:08
@shoumikhin
shoumikhin force-pushed the shoumikhin/drop-coreml-static-int8-operators branch from 3b0dd40 to ad459d8 Compare August 10, 2026 18:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JakeStevens
JakeStevens merged commit 6766736 into main Aug 11, 2026
197 checks passed
@JakeStevens
JakeStevens deleted the shoumikhin/drop-coreml-static-int8-operators branch August 11, 2026 14:45
JakeStevens pushed a commit that referenced this pull request Aug 11, 2026
…#21691)

## What is broken

Every so often a commit on main shows a cancelled ARM or CoreML job on
HUD even though nobody cancelled anything and the commit is fine. It
looks like a real failure, so the oncall investigates a phantom.

## Why it is broken

GitHub Actions cancels an older run when a newer run lands in the same
concurrency group. These workflows use a group key built like this:

```
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}
cancel-in-progress: true
```

The key already tells apart pull requests, branches, commits and manual
runs. It does not tell apart a scheduled run.

These same workflows also have a nightly `schedule` trigger. When the
nightly cron fires, `github.ref_name` is `main` and `github.sha` is the
current head of main, which are exactly the same values the push run for
that commit used. Same key, `cancel-in-progress: true`, so the cron run
kills the push run that was still going.

The result is a commit whose CI reads as cancelled with no explanation.

## The fix

Add `-${{ github.event_name == 'schedule' }}` to the concurrency group,
so a scheduled run and a push run on the same commit sit in different
groups and no longer cancel each other.

This is the same pattern already used by other workflows in this repo.
It is applied here to every workflow that has both a `schedule` and a
`push` trigger:

- `test-backend-arm.yml`
- `test-backend-coreml.yml`
- `test-backend-openvino.yml`
- `test-backend-qnn.yml`
- `test-backend-vulkan.yml`
- `test-backend-webgpu.yml`
- `test-backend-xnnpack.yml`
- `test-webgpu-native.yml`
- `build-cmsis-pack.yml`
- `docker-builds.yml`

`doc-build.yml` also has both triggers but is deliberately left alone:
it sets `cancel-in-progress: ${{ github.event_name == 'pull_request'
}}`, so outside a pull request it never cancels anything and the
collision cannot happen there.

This only separates schedule from push. It does not change how two
scheduled runs
of the same workflow behave: `github.sha` is still in the key, so two
scheduled
runs share a group only while `main` has not moved between them, and
that is
unchanged by this PR.

## How this was verified

Every workflow under `.github/workflows` with a `schedule:` trigger was
listed
together with its concurrency group, to make sure the change is both
correct and
complete:

- `apple.yml`, `periodic.yml` and `riscv64.yml` already carry exactly
the
`-${{ github.event_name == 'schedule' }}` suffix this PR adds, so this
is not a
new idea, it is existing practice in this repo. `periodic.yml` goes one
step
further and also appends `${{ github.event.schedule }}` to separate its
  individual crons.
- The 10 workflows changed here are precisely the ones still on the
older key
  that also have a `push` trigger, so none were missed.
- `build-cadence-runner.yml` and `test-pico2-build.yml` already put
`${{ github.event_name }}` in the key, which covers the same collision.
- `nightly.yml` has both triggers but only pushes on `ciflow/nightly/*`
tags, so
its `github.ref_name` is a tag and can never match the scheduled run on
`main`.
- `doc-build.yml` is the case described above.
- The other scheduled workflows have no `concurrency` block at all.

Note that the fix only takes effect once it is on `main`, because a
scheduled run
always uses the workflow file from the default branch.

CI on this pull request: 213 checks reported, 195 success, 15 skipped, 2
cancelled, 0 failures. Both cancellations were manual, and in one of
them every
step had already finished successfully before the cancel landed.

## Overlap with other pull requests

Two of the files here are also touched by other open pull requests, in
both cases
far away from the `concurrency:` block this PR edits:

- `.github/workflows/test-webgpu-native.yml` also gets a `timeout:`
change in
  #21690, in the job block further down.
- `.github/workflows/test-backend-coreml.yml` also gets a matrix
`exclude:` entry
  in #21695, likewise further down.

The hunks do not touch and the changes merge cleanly in any order.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants